Welcome to the activation-interventions module. Probes can show that information is readable from an activation, but they cannot by themselves establish what the model does with that information. This module turns to interventions: changing internal values, observing the resulting behavior, and specifying exactly which causal claim the change supports.
This lesson focuses on a deceptively simple choice. If you replace a component’s activation to see whether it matters, what do you replace it with? In about 40 minutes, you will compare zero, mean, and resampled ablations, implement all three at one hook site, and interpret disagreements between their estimated effects.
An ablation is a comparison, not an eraser
Suppose is the output of one attention head or MLP on input , and is a score for the behavior you care about. You might use the logit difference between a desired answer and an alternative; a larger score would then mean stronger preference for the desired answer.
An ablation substitutes a value for , reruns the downstream computation, and measures the difference:
Here, means the same model with that particular site replaced by . A positive means the replacement reduced the target-behavior score. It does not yet mean that the component is necessary in every context. The measured effect depends on the input, the site and position, the score, and the replacement.
Read the three baseline descriptions before working through an example.
Activation Patching and Causal Interventions | Learn Mechanistic Interpretability
Read the “Ablation: Choosing a Baseline” section of Learn Mechanistic Interpretability for the distinct interpretations of zero, mean, and resampled replacements.
Under “Ablation: Choosing a Baseline,” start at the opening sentence about substituting an activation from one run into another. Read three baselines, stopping at the sentence about needing multiple resamples, before the discussion of causal scrubbing. Focus on what each replacement preserves, what it removes, and why a replacement drawn from real activations can still form an unusual combination with the current input.
The three choices correspond to three different counterfactuals:
- Zero ablation: The site emits zero. For a component whose output is added to the residual stream, this removes its additive update. At a hidden site inside a nonlinear computation, zero may instead encode a particular, meaningful state.
- Mean ablation: The site emits its average activation over a specified reference set. This retains an average contribution while removing variation at that site across inputs. Other parts of the model still receive the current input.
- Resampled ablation: The site emits an activation recorded on another input. Each replacement occurred naturally somewhere, although the donor activation and the current input may not have occurred together.
None is a universal “no information” value. In particular, a dataset mean need not itself be a typical activation, and an empirical donor value need not be typical in the recipient’s context.
Why the estimates can disagree
Consider a one-site toy model with activation and downstream score
For the target input, , so the intact score is . A separate reference set has activations ; its mean is .
| Replacement | Resulting score | Drop from intact score |
|---|---|---|
| None: | ||
| Zero: | ||
| Mean: | ||
| Resample: |
The average signed drop under these four equally likely resamples is . Thus the same target input and intervention site yield drops of , , and under zero, mean, and average resampling.
The mean and resampled estimates differ despite using the same reference set. The downstream ReLU makes the score at the mean activation, , different from the mean score across donor activations, . If the downstream score were linear in this activation, those two expected scores would agree for this donor distribution.
Notice also the donor value : replacing with raises the score, giving a negative drop. Averaging signed effects answers whether the replacement tends to lower the score; it can conceal donor-to-donor variability. Reporting the donor-effect distribution alongside its average makes the estimate more informative.
A small hook-based comparison
Because you already know PyTorch hooks, the useful discipline here is to keep the site, recipient input, and metric fixed while changing only the replacement rule. The following self-contained model reproduces the table. Its component is an identity module so that the hook has a clearly defined activation to replace.
import torch
from torch import nn
class ToyModel(nn.Module):
def __init__(self):
super().__init__()
self.component = nn.Identity()
def forward(self, a):
return torch.relu(self.component(a) - 1).squeeze()
model = ToyModel().eval()
target = torch.tensor([[3.0]])
reference_inputs = torch.tensor([[0.0], [0.0], [2.0], [6.0]])
# Cache activations on a reference set, not on the target run.
cached = []
def cache_hook(_module, _inputs, output):
cached.append(output.detach().clone())
handle = model.component.register_forward_hook(cache_hook)
try:
with torch.no_grad():
model(reference_inputs)
finally:
handle.remove()
donor_pool = cached[0] # Shape: [4, 1]
mean_activation = donor_pool.mean(dim=0, keepdim=True)
def score_with(replacement=None):
if replacement is None:
with torch.no_grad():
return model(target).item()
def replacement_hook(_module, _inputs, output):
return replacement.expand_as(output).clone()
handle = model.component.register_forward_hook(replacement_hook)
try:
with torch.no_grad():
return model(target).item()
finally:
handle.remove()
intact = score_with()
zero_score = score_with(torch.zeros_like(mean_activation))
mean_score = score_with(mean_activation)
donor_scores = [
score_with(donor_pool[i:i + 1])
for i in range(len(donor_pool))
]
print("intact:", intact)
print("zero drop:", intact - zero_score)
print("mean drop:", intact - mean_score)
print("resampled drops:", [intact - value for value in donor_scores])
print("average resampled drop:",
sum(intact - value for value in donor_scores) / len(donor_scores))
The expected drops are 2.0, 1.0, and an average resampled drop of 0.5. In a transformer, the same comparison requires care about where the hook sits. Replacing an MLP’s output with zero removes that MLP’s update at the selected location; replacing an entire residual-stream state with zero also removes information contributed by earlier components. For a module returning a tensor with shape [batch, position, hidden], replacing one token position typically means cloning the output and modifying only patched[:, position, :]. A module that returns a tuple requires handling its output structure rather than treating it as a tensor.
Designing an interpretable comparison in a real model
First fix the behavior and its measurement. For a two-answer language-model task, a logit difference can track gradual changes that accuracy would hide. Keep the same candidate answers and scoring convention when evaluating all three ablations on a given recipient. Also track an off-target performance metric: a large target-score drop is less persuasive as evidence of a specific contribution if the intervention broadly damages the model.
Next define the activation site precisely: component, layer, token position, and whether the hook captures a component update or a shared residual state. Use the same site for all baselines. Evaluate the model deterministically, remove hooks after each run, and recompute all downstream activations after substitution.
Finally define the reference distribution before seeing which baseline gives the preferred result. A mean from unrelated text, a mean from task-matched examples, and a mean from examples sharing the recipient’s answer can answer different questions. Donor selection matters just as much for resampling. If donors systematically carry the correct task signal, resampling may preserve it; if they carry opposing signals, it may strongly disrupt it. Draw multiple donors per recipient, record their individual effects, average within each recipient, then compare results across recipients. Do not treat every donor draw from the same recipient as an independent test case.
A useful report therefore states not just “ablating this component lowers the score,” but something like: At this MLP output and token position, zero replacement lowered the target logit difference more than a task-matched mean did; resampled effects varied substantially across donors. That wording identifies the evidence without pretending the replacements are interchangeable. Agreement across sensible baselines strengthens a claim that behavior depends on the site under those interventions. Disagreement is also a result: it directs attention to constant contributions, nonlinear downstream responses, donor content, or unusual hybrid states.
Key takeaways
An ablation effect is relative to its replacement baseline. Zero removes an additive component update when the site is chosen appropriately; the mean removes its input-dependent deviation from a chosen reference average; resampling substitutes another input’s activation and produces a distribution of effects. None is automatically neutral or guaranteed to keep the whole model state in-distribution.
In the next lesson, you will replace activations using matched reference and counterfactual inputs rather than a generic baseline. That makes the substituted value’s content, and the behavioral change it predicts, more specific.