Skip to main content

Welcome to a Grasp lesson

Create a free account to keep studying and create your own study path.

Lesson 30

Evaluating Shortcut Mechanisms with Counterfactuals and Internal Interventions

Recommended resources

The previous lesson established an input-level result: when the same review appears under different source headers, the model’s sentiment score may change even though the review and its correct label do not. That tells you the model is sensitive to the header. It does not tell you which internal computation carries that influence.

This lesson tests the next, narrower claim: a proposed feature or circuit helps transmit the header’s effect to the prediction. You will compare the measured input-level effect with targeted activation swaps, then assess whether the swaps support the proposed mechanism rather than merely changing the output.


State the mechanism as a prediction

Keep the matched review pairs from the previous lesson. Let be review under a header correlated with positive sentiment, and the same review under a neutral or differently correlated header. Define a signed score from the model’s output logits:

The input-level effect for a pair is . Averaging this difference across held-out reviews estimates how strongly changing the header shifts the model toward positive sentiment on those examples.

Now consider a more specific hypothesis: an attention head reads information about the header and writes a positive-sentiment signal at the classification position. This predicts more than a header-sensitive score or an attention pattern pointing at the header. If that head carries part of the relevant signal, replacing its activation from one header condition with its activation from the other should shift the score in a predictable direction.

The video introduces the idea of testing a proposed internal mediator and, in its addition example, shows why a mechanistic hypothesis should predict the result of a particular internal swap—not just a change in the final answer.

Causal Mechanistic Interpretability (Stanford lecture 1) - Atticus Geiger

Watch “Causal Mechanistic Interpretability (Stanford lecture 1)” by Atticus Geiger on Goodfire for the logic of connecting an input effect to an internal variable.

Watch mediation setup for the distinction between changing an input and changing a proposed mediator. Then watch carry intervention: focus on how the proposed location of a carry bit yields a specific counterfactual prediction. You need not adopt the addition example as a model of sentiment classification; use its standard of specificity for the header hypothesis.

A prediction should identify the site (for example, one head’s output at the classification position), the replacement (the corresponding activation from the other header condition), and the expected score change. Selecting a head because its attention visualization looks suggestive is a way to generate that hypothesis, not to confirm it.



Put the input and internal interventions on the same scale

Activation patching runs the model on two inputs, stores an activation from one run, and substitutes it into the other. The recipient run then recomputes everything downstream of the patched site. The image illustrates this operation with two factual prompts: an activation from one run is inserted into the other, and the output changes. For our shortcut test, the crucial difference is that both inputs retain the same task evidence and gold label; only the suspected nuisance cue changes.

An activation from the Colosseum prompt’s run is inserted into the Eiffel Tower prompt’s run, shifting the displayed output toward “Rome.” The diagram illustrates a targeted activation swap, not proof that a single feature exclusively carries an input effect.

The following reading supplies the clean/corrupted patching procedure and explains why the direction of a swap matters. Here, “clean” and “corrupted” are merely conventional names for the two runs: neither header version is inherently the correct sentiment input.

Activation Patching and Causal Interventions | Learn Mechanistic Interpretability

Read “Activation Patching and Causal Interventions” from Learn Mechanistic Interpretability to connect the two-input setup to the causal question answered by each swap.

In “The Clean/Corrupted Framework,” start at “The basic setup requires two model runs” and read the patching setup through the explanation of why a non-effect does not settle a component’s role. Then, in “Noising vs. Denoising,” read both directions. Focus on what the recipient model computes after each replacement, rather than treating either direction as an unconditional proof of necessity or sufficiency.

Suppose and are the selected head outputs cached from the two runs. Write for the score obtained by running , substituting at that site, and recomputing downstream. Two informative interventions are:

If this head transmits a positive shift associated with header A, both quantities are predicted to be positive. The first asks whether installing the B-run activation reduces A’s effect in an A-run context. The second asks whether installing the A-run activation introduces a positive shift in a B-run context. Neither tests the head in isolation: the rest of the recipient run remains present.

For one illustrative review, suppose the unpatched scores are under A and under B, an input-level difference of . Replacing the selected activation in the A run with its B value yields ; replacing it in the B run with its A value yields . The respective changes, and , have the predicted signs and are substantial relative to the observed input difference. They are not two additive shares of a fixed : the swaps use different recipient contexts, and interacting or redundant pathways can affect both measurements.

Use the same signed score, review pairs, and classification position for the input and patching results. If header changes alter token lengths, do not silently assume token positions align. Specify the corresponding site in each run, or construct header variants whose relevant positions align. A whole residual-stream replacement may change the answer while revealing little about a proposed head-specific mechanism.



Compare a mechanism claim with an actual shortcut study

A text-classification study offers a useful example of moving from an observed shortcut to a proposed route through the model. Its nuisance cue is an actor name inserted into a review, rather than our source header. The investigators report heads that directly influence label logits, then test which earlier components influence the prediction through those heads. That second step matters: finding an influential output head alone does not identify where its cue-related signal came from.

Mechanistic Investigation of Shortcuts in Text Classification

Read the method and initial patching results in “Mechanistic Investigation of Shortcuts in Text Classification.” Treat the actor-name shortcut as a comparison case, not as evidence about the hypothetical source-header model.

On p. 107, in the “Path Patching” paragraph immediately before Section 4, “Classification under Shortcuts,” read path-patching method. In Section 5.2, “Patching Results” (p. 109), read from “Figure 5a demonstrates the results” through the discussion of Figure 5a’s right-hand panel, ending where it says later MLP layers do something similar. Start with direct effects; then follow the paragraph beginning “Next, we investigate which preceding components” to the right-hand panel. Notice why the direct-output and through-head tests identify different parts of the proposed route.

This distinguishes node patching from a more specific path-patching claim. Swapping a head output and observing a score change implicates that activation under the test conditions. Constraining which downstream component receives a patched signal tests a proposed connection. Neither measurement, on its own, establishes that the proposed route is the model’s only route for using the cue.

The study also tests whether transferring its identified circuit into a comparison run recreates some shortcut behavior, and whether ablating identified heads reduces the shortcut effect. Read those results with the intervention and metric in view:

Mechanistic Investigation of Shortcuts in Text Classification

Continue the same paper to see how the authors check their proposed circuit against behavior and what remains unresolved after an ablation.

On p. 110, after “The Shortcut Mechanism” and before Section 6, read the paragraph beginning “To validate the faithfulness of the shortcut circuit” through the transfer result, and inspect Table 1 directly below it. Then find Section 7, “Shortcut Mitigation” (p. 112): read the ablation result, including Table 2, stopping before Section 8, “Qualitative Analysis.” Track both the reduction in shortcut effects and the authors’ qualification about compensation.

In Table 1, circuit transfer reproduces part, not all, of the accuracy change associated with the actor-name cue: the reported recovery is about for one class condition and for the other. These are recoveries of a specified accuracy contrast under a specified patch, not percentages of neurons devoted to the shortcut or a unique partition of its causal effect. The later ablation reduces the measured shortcut effect, providing a complementary test, while leaving open whether other components compensate or the same heads serve legitimate predictions.



Decide what the combined evidence warrants

The two experiment types answer different questions. Input-level counterfactuals establish whether the nuisance edit changes behavior while task evidence stays fixed. Internal swaps test whether changing a proposed mediator affects that behavior under the chosen recipient context. To connect them, evaluate a proposed head on held-out matched pairs and compare its patch effects with the original paired score differences—not just with a few visually compelling examples.

Several checks sharpen the comparison:

  • Directional prediction: On reviews where A raises the positive score, does substituting the B activation usually lower it? Does the reverse swap usually raise the B score? Report the distribution across reviews, including contrary cases, rather than only an average recovery.
  • Specificity: Compare with swaps at plausible alternative sites and with a same-cue donor matched as closely as practical. If many unrelated swaps produce similar shifts, a damaged computation is a competing explanation.
  • Task preservation: Check behavior on examples without the suspected cue, or on matched examples where the cue is irrelevant to the hypothesis. A patch that changes sentiment scores indiscriminately is weak evidence for a selective header mechanism.
  • Generalization: Identify candidate sites on one set of reviews and estimate effects on separately held-out reviews, spanning both gold labels and varied wording. Assess uncertainty at the level of independent base reviews.

A positive result supports a conditional causal claim: at this site, with these donors, recipients, inputs, and score, the replacement changes the model’s response in the predicted way. It does not by itself show that the head encodes only source identity, that it normally acts alone, or that the measured recovery is a unique fraction of the header’s effect. Conversely, a near-zero patch does not rule out cue use elsewhere: the selected site may be wrong, another route may be redundant, or downstream computation may compensate.

The strongest interpretation comes from agreement between a well-checked input contrast, a pre-specified internal prediction, suitably specific intervention effects, and behavior on held-out examples. Disagreement is informative too. A reliable input effect with no specific patch effect argues against the tested mechanism, not against the behavioral shortcut already observed.



Takeaway

A shortcut mechanism must explain how an established input-level cue effect travels through the model. Matched counterfactuals measure that effect; targeted swaps test a proposed mediator against it. Report the patch site, donor and recipient conditions, direction, metric, controls, and scope of the result before calling an activation or circuit causal.

The next lesson examines a further constraint on mitigation: suppressing a shortcut-associated feature may also harm legitimate behavior that uses the same representation.