Skip to main content

Welcome to a Grasp lesson

Create a free account to keep studying and create your own study path.

Lesson 19

Comparing Individual and Joint Ablations for Redundancy and Non-Additive Effects

Recommended resources

Welcome back. In the previous lesson, you isolated a proposed connection between two transformer components by changing what a receiver could obtain from a sender. Now the question shifts from a connection to a set of components: if two heads appear to support the same answer, what happens when you remove each one separately, and what happens when you remove both?

That comparison can reveal contributions that a one-head-at-a-time study misses. It can also reveal a trap: a large joint effect does not, by itself, tell you why the components interact.


Why one ablation can miss a mechanism

The indirect-object identification (IOI) circuit provides a useful setting. In the diagram below, several head classes contribute to selecting the appropriate name. Focus on the output stage: Name Mover Heads and Backup Name Mover Heads both appear there.

Schematic of an indirect-object identification circuit. Earlier head classes feed S-Inhibition Heads, while Name Mover, Backup Name Mover, and Negative Name Mover Heads participate near the output. The diagram proposes relationships; it does not establish their effects under ablation.

The word backup suggests a testable possibility. A primary head may make a substantial contribution in an ordinary run, yet removing it may have only a modest effect if another head contributes more in the altered run. The diagram cannot establish that response. You need to compare runs.

Evaluating Circuits: Faithfulness and Completeness | Learn MI

Read the two IOI case studies in Learn MI's “Evaluating Circuits: Faithfulness and Completeness.” Together they show why a small or even beneficial single-component ablation need not mean a component is irrelevant.

In “Negative Name Movers: Components That Work Against the Task,” read the negative contribution account. Then, in “Backup Name Movers: Built-In Redundancy,” start with the paragraph beginning “When primary Name Mover Heads were ablated” and read the backup discovery. Focus on the difference between a head’s contribution in an ordinary run and its contribution after another head is removed.

A Negative Name Mover illustrates one complication: removing a component can improve the selected task metric. Backup Name Movers illustrate another: the effect of removing one component depends on which others remain available. Neither phenomenon can be diagnosed reliably by sorting heads according to their individual ablation scores.



The four-run comparison

Choose two components, and , a held-out batch of prompts, and one target metric. For an IOI-style next-token task, let be the logit for the correct name minus the logit for the competing name. Higher means stronger preference for the correct answer.

Run the same prompts under four conditions:

Condition Measured margin
Neither component ablated
Only ablated
Only ablated
Both ablated in one run

Define each ablation’s loss of margin relative to the intact model:

A negative means that ablating improved this metric. To test whether the joint loss is simply the sum of the two individual losses, compute

This is an interaction under the specified intervention:

  • near zero: the measured losses are approximately additive.
  • : the joint ablation hurts more than the individual losses predict.
  • : the joint ablation hurts less than their sum predicts.

For example, suppose the four margins are , , , and . The individual losses are and , but the joint loss is , giving . Looking only at the individual losses would miss most of the vulnerability exposed by removing both heads.

Joint ablation means intervening on both components in the same forward pass and allowing unaffected downstream computation to run. It does not mean adding two scores from separate runs. Nor should you insert a cached value for from the intact run after ablating : that would prevent from responding to the change, and would answer a different causal question.

What does an interaction tell you?

A simple parallel-path toy model clarifies the positive case. Suppose either or can sustain a margin of . Then , but . The joint effect is large despite both single effects being zero. That is redundancy: either remaining path suffices in this toy model.

For contrast, suppose both components are required together, and the margin is only while both operate. Then and . Each single ablation already destroys the behavior; removing the other cannot destroy it again. Here . This overlapping-loss pattern is consistent with a serial dependency, but does not prove one.

The same numerical patterns can arise for other reasons. Downstream nonlinearities may change the output’s sensitivity to two removed contributions. A later head might compensate for an earlier ablation. Even the definition of “removed” matters: replacing a head output with zero can have different consequences from replacing it with a mean or a matched activation. Treat as evidence of non-additivity to explain, not a direct readout of circuit architecture.

Activation Patching and Causal Interventions | Learn Mechanistic Interpretability

Read Learn Mechanistic Interpretability’s discussion of self-repair to connect a measured interaction with possible downstream responses—without assuming that every interaction is an intelligent backup mechanism.

In “Confounds: Self-Repair and Compensation,” read the self-repair discussion through the end of that subsection. Distinguish a backup component changing its output from effects such as LayerNorm rescaling; both can alter the final ablation score.

If is earlier than , you can investigate a backup hypothesis more directly: compare ’s output in the intact run with its output when is ablated, then test whether also ablating removes the remaining behavior. An increased output norm alone is not proof that carries the relevant answer information; the joint behavioral test supplies an additional check. If two heads compute in parallel within the same layer, one generally cannot respond within that layer to the other’s output ablation, although later components can respond to their changed combined contribution.



Implementing the comparison with head-output hooks

The scaffold below uses the post-output-projection result hook from the previous lesson. It zeroes selected heads’ additive residual-stream contributions at all sequence positions, leaving other heads free to recompute. Keep the ablation site, positions, and replacement rule identical across the four conditions.

import torch
from transformer_lens import utils

# Supply a studied pair of (layer, head) indices.
A = (9, 9)
B = (10, 1)

# Supply a batch of held-out prompts and one-token candidate IDs per prompt:
# clean_tokens: [batch, sequence]
# good_ids, bad_ids: [batch]
model.eval()
model.set_use_attn_result(True)

batch_idx = torch.arange(
    clean_tokens.shape[0], device=clean_tokens.device
)

def margin_with_ablations(ablated):
    # Several selected heads may share one layer's result hook.
    heads_by_hook = {}
    for layer, head in ablated:
        name = utils.get_act_name("result", layer)
        heads_by_hook.setdefault(name, set()).add(head)

    def zero_selected_heads(value, hook):
        patched = value.clone()
        selected = sorted(heads_by_hook[hook.name])
        patched[:, :, selected, :] = 0.0
        return patched

    hooks = [
        (name, zero_selected_heads)
        for name in heads_by_hook
    ]
    with model.hooks(fwd_hooks=hooks):
        logits = model(clean_tokens)

    return (
        logits[batch_idx, -1, good_ids]
        - logits[batch_idx, -1, bad_ids]
    )

with torch.inference_mode():
    m0 = margin_with_ablations(set())
    mA = margin_with_ablations({A})
    mB = margin_with_ablations({B})
    mAB = margin_with_ablations({A, B})

dA = m0 - mA
dB = m0 - mB
dAB = m0 - mAB
interaction = dAB - dA - dB

print("Mean margins:", [
    x.mean().item() for x in (m0, mA, mB, mAB)
])
print("Mean individual and joint losses:", [
    x.mean().item() for x in (dA, dB, dAB)
])
print("Mean interaction:", interaction.mean().item())

The tensors retain a separate score for each prompt. Inspect that distribution alongside its mean: a large average interaction might be driven by a small subset of examples, and effects of opposite signs can cancel. Use the same matched examples and candidate-token definitions in every condition. As in the previous lesson, check individual answer logits as well as their difference, and retain an off-target performance metric so that a task-specific improvement is not mistaken for an overall improvement.

Zeroing a head’s result removes its additive write at this hook site, not the head’s parameters or every influence associated with it. If you repeat the comparison with mean or resampled replacements, specify how replacement values were obtained and use a consistent rule across all four runs. Different baselines may produce different interactions; that sensitivity is itself worth reporting.



From a joint effect to a defensible claim

A useful report states the four measured margins—not just —and names the intervention, metric, prompt distribution, and component sites. You can then separate observations from explanations:

“On these held-out prompts, zero-ablating either head alone changed the correct-name margin little, while zero-ablating both caused a large drop.”

That is a supported behavioral interaction. “The second head is a backup for the first” is a further mechanistic claim. It becomes more credible if the proposed backup’s relevant contribution increases after the primary ablation, their joint removal eliminates the remaining behavior, and the pattern persists across matched prompts and reasonable replacement baselines. It remains conditional on those tests; other downstream responses may contribute.

Individual ablations answer what happens when one component is removed while the rest of the model can respond. Joint ablations reveal what those responses may have concealed, while the interaction score describes—not explains—the resulting non-additivity. In the next lesson, you will widen the evaluation from a pair of components to a candidate circuit and ask how faithfully it reproduces the full model’s behavior on held-out examples.