Skip to main content

Welcome to a Grasp lesson

Create a free account to keep studying and create your own study path.

Lesson 17

Ranking Components with Gradient-Based Attribution Patching

Recommended resources

Hello. This module moves from testing individual activation interventions to investigating circuits: collections of components that may jointly produce a behavior. You have already encountered the central experimental ingredients—matched inputs, a target-behavior metric, and hooks that let you replace an activation. The difficulty now is scale. If a model has hundreds of attention heads and MLP components, which ones merit an exact intervention?

This lesson develops attribution patching as a fast first screen. You will use gradients to estimate how much replacing each candidate component’s corrupted activation with its clean activation would change a metric, rank the candidates, and then decide what to patch exactly. The distinction to keep in view throughout is that a gradient-based score is a prediction about an intervention, not the result of performing it.


The intervention we want to approximate

Suppose a clean prompt supports a desired answer, while a matched corrupted prompt changes one relevant cue. Choose a metric such as the logit of the desired answer minus the logit of a foil answer, measured at a fixed output position. Use the same metric definition on both prompts. A larger value then consistently means more support for the desired answer.

For candidate component , let and be its activations on the clean and corrupted inputs. Let mean: run the corrupted input, replace component ’s activation with , and measure the resulting output metric. The exact patch effect is

That definition holds the receiving run fixed. It measures what happens when this particular clean activation is inserted into this particular corrupted computation—not whether the component is universally necessary, or whether it has a single interpretable function.

Attribution Patching: Activation Patching At Industrial Scale

Read Neel Nanda’s explanation for the central approximation and, especially, for why he treats it as a search tool rather than a final circuit explanation.

In “What Is Attribution Patching?”, read the core idea, beginning with the opening paragraph and stopping after the sentence about two forward passes and one backward pass. The formal notation that follows is optional; the lesson supplies notation below. Then, in “Why Care?”, start at “The main reason I think you should care about this is that attribution patching is really fast and scalable!” Read the search rationale through the recommendation to check top candidates with actual patching. Skim the asymptotic-runtime aside; focus on the difference between cheaply ranking candidates and verifying a claim.

Attribution patching replaces an expensive experiment with a first-order Taylor approximation at the corrupted activation:

The two factors answer different questions. The activation difference says what would be inserted. The gradient with respect to that activation, evaluated on the corrupted run, says how sensitive the chosen metric is to a small change at that site. Their dot product estimates the effect of inserting the whole difference. This is an activation gradient, not a gradient with respect to the model’s weights; no training update occurs.

For a head activation with multiple positions and channels, the dot product sums products over whichever dimensions belong to the proposed patch. If the experiment replaces one head at all positions, sum over positions and channels. If it replaces that head at just one aligned position, sum only over channels there. The dimensions you sum over must describe the same intervention you later test exactly.

A positive score predicts that the clean activation will raise the desired-answer logit gap on the corrupted input. A negative score predicts that it will lower it; negative does not mean “unimportant.” For a broad search for components that influence the metric, rank by absolute score while preserving each score’s sign. If the specific hypothesis concerns restoring the desired answer, prioritize positive scores—but retain some negative and near-zero candidates for diagnostic exact tests.



One backward pass, many candidate scores

Why is this economical? A clean forward pass supplies . A corrupted forward pass supplies and the metric. Backpropagating that metric through the corrupted run supplies gradients at all retained candidate activations. You can then compute a separate dot product for every head without another model run. Exact patching, by contrast, requires a new forward run for each separately tested candidate.

Attribution Patching - nnsight

The nnsight tutorial shows this caching pattern on transformer attention heads. Read it to connect the mathematical score to hookable tensors and a layer-by-head ranking.

In “Attribution Patching Over Components,” start at the sentence “We take the gradient of the patch metric” and read the head example through the interpretation immediately after the head heatmap. Follow the clean cache, corrupted cache, backward pass, and reduction into head scores. The example hooks the input to an attention output projection because that model does not expose a separate hook module for each head. Attend to how its tensor is partitioned into heads before interpreting any heatmap cell.

Here is the same procedure in ordinary PyTorch. It assumes a model with model.blocks, where each block has a head_hook module placed after per-head attention outputs have been formed and before they are combined. Each hook receives a tensor shaped [batch, position, head, channel]. This is an interface to adapt to your model, not a claim that standard transformer implementations expose head_hook by default. Keep dropout disabled and make sure the clean and corrupted examples have aligned positions.

import torch

model.eval()
sites = [block.head_hook for block in model.blocks]

# target and foil: [batch] token IDs, fixed for each clean/corrupted pair
def gap(logits, target, foil):
    batch = torch.arange(logits.shape[0], device=logits.device)
    return logits[batch, -1, target] - logits[batch, -1, foil]

clean_cache = {}
corrupt_cache = {}

def save_clean(layer):
    def hook(module, inputs, output):
        clean_cache[layer] = output.detach().clone()
    return hook

handles = [site.register_forward_hook(save_clean(i))
           for i, site in enumerate(sites)]
with torch.no_grad():
    clean_logits = model(clean_tokens)
for handle in handles:
    handle.remove()

def retain_corrupt(layer):
    def hook(module, inputs, output):
        output.retain_grad()
        corrupt_cache[layer] = output
    return hook

model.zero_grad(set_to_none=True)
handles = [site.register_forward_hook(retain_corrupt(i))
           for i, site in enumerate(sites)]
corrupt_logits = model(corrupt_tokens)
baseline_gap = gap(corrupt_logits, target, foil).detach()

# Sum, rather than mean, so each example's gradient is unscaled.
gap(corrupt_logits, target, foil).sum().backward()
for handle in handles:
    handle.remove()

batch_size = corrupt_tokens.shape[0]
scores = torch.stack([
    (
        corrupt_cache[i].grad
        * (clean_cache[i] - corrupt_cache[i].detach())
    ).sum(dim=(0, 1, 3)) / batch_size
    for i in range(len(sites))
])  # [layer, head]: mean signed attribution over the batch

num_heads = scores.shape[1]
flat_indices = scores.abs().flatten().topk(k=6).indices
candidates = [divmod(index.item(), num_heads) for index in flat_indices]

The code sums over batch, position, and channel, leaving one score per head per layer. Using the sum of per-example metric values for backpropagation, followed by division by batch size, makes each reported score a mean per-example attribution. If you instead backpropagated a batch mean and divided again, your scores would be incorrectly scaled.

Before trusting the ranking, check that the clean and corrupted metric values actually differ on the chosen pairs. Also check that head_hook captures the intended per-head tensor: in a fused implementation, a tensor merely shaped like head outputs need not be located at a site where replacing one slice implements the patch you have in mind.

Turning the ranking into exact tests

The next action is not to name the top head a “circuit.” It is to run the corrupted input again, replacing only that head’s slice with the corresponding clean slice, and compare the observed metric change with its score. In the model interface assumed above, a single-candidate test looks like this:

def exact_head_effect(layer, head):
    def patch(module, inputs, output):
        patched = output.clone()
        patched[:, :, head, :] = clean_cache[layer][:, :, head, :]
        return patched

    handle = sites[layer].register_forward_hook(patch)
    try:
        with torch.no_grad():
            patched_logits = model(corrupt_tokens)
        per_example_effect = (
            gap(patched_logits, target, foil) - baseline_gap
        )
        return per_example_effect.mean().item()
    finally:
        handle.remove()

comparison = [
    (layer, head, scores[layer, head].item(),
     exact_head_effect(layer, head))
    for layer, head in candidates
]

Compare signed attribution with signed exact effect, not only their magnitudes. Test the highest-ranked candidates, and sample some lower-ranked candidates as a check against missed effects. Then repeat promising comparisons on held-out matched pairs. The forward hook in the exact test returns a modified tensor; the hooks in the screening pass only observe and retain tensors. That operational difference is the boundary between an estimated patch and an intervention.



What a ranking can miss

A first-order score measures the slope at the corrupted state. It need not describe the whole trip to the clean activation. Consider a toy metric , with corrupted activation and clean activation . The gradient at the corrupted state is zero, so attribution predicts no effect; exact patching changes the metric by one. Conversely, a steep local slope can overpredict an effect if the metric flattens along the patch direction.

Real networks introduce such departures from linearity through mechanisms including nonlinear MLP activations, attention softmax, and normalization. A second problem is interaction: two components may jointly matter even if replacing either one alone has little effect in the corrupted context. Individual attribution scores do not establish what a joint intervention would do.

Attribution Patching: Activation Patching At Industrial Scale

Return to Nanda’s post for evidence about which patch sizes make the approximation more or less reliable. Treat its reported results as findings for the studied setting, not a guarantee for every model.

In “Does This Work In Practice? (Experiments),” read the reported pattern in the opening paragraph. Continue through “Big activations,” from its opening comparison to the paragraph discussing why residual-stream changes can be difficult to approximate, then read “Zooming in on Heads” for the finer-grained comparison. In “Comparing ranks,” read the ranking criterion. Focus on why a method can be useful for selecting tests even when its numerical effect estimates are imperfect.

This suggests a practical screening policy. Prefer well-defined, relatively targeted candidate patches; specify the metric, direction, positions, and component boundary before ranking; and compare estimates with exact effects on more than just the apparent winners. A low score is not proof of irrelevance—false negatives are particularly costly when screening decides what receives further investigation. A high score is a reason to test, not evidence that you have recovered a complete mechanism.



Takeaway

Gradient-based attribution patching uses a clean–corrupted activation difference and a gradient from the corrupted run to rank proposed component patches cheaply. Its speed comes from reusing two forward runs and one backward pass across many candidates. Its limitation comes from using a local linear estimate for a finite, potentially nonlinear intervention.

Keep the resulting heatmap in its proper role: a map of where to spend exact-intervention effort, with signs, patch boundaries, and failed predictions recorded alongside successes. The next lesson narrows the question from “which component matters?” to “does a hypothesized connection between two components carry the effect?” using edge-specific patching.