Skip to main content

Welcome to a Grasp lesson

Create a free account to keep studying and create your own study path.

Lesson 13

Evaluating Conditional Necessity and Sufficiency in Intervention Results

Recommended resources

Last lesson, you patched activations between matched inputs and measured restoration in one direction and disruption in the other. In the hand-set toy network, the cue-position patch worked both ways. A result like that establishes that the substitution changed the output; interpreting it as necessity or sufficiency requires one more step: specify the behavior, the replacement, and the state of the rest of the model.

This lesson gives you a precise way to make that judgment. The distinction matters when investigating a suspected shortcut: showing that a shortcut-associated activation can steer an answer is not the same as showing that the unmodified model needs it to produce that answer.


Define the behavior before naming the result

Let the clean input produce a behavior of interest and the matched corrupted input not produce it. “Clean” and “corrupted” name their roles in the experiment; they do not necessarily mean grammatical versus damaged text. Both inputs can be ordinary, valid prompts.

Choose a fixed measure of the behavior. For example, a logit difference might measure preference for answer A over answer B on every run. If the claim is about a categorical behavior—such as preferring A—decide what counts as exhibiting it, perhaps a positive logit difference. If the claim is about the strength of that preference, report the margin as well. A patch that raises the margin substantially but leaves it negative has a causal effect on the score without restoring the stated categorical behavior.

At a selected site, write for the activation recorded on the clean run and for the one recorded on the corrupted run. The two relevant experiments have different recipients:

The notation after the semicolon means that the run proceeds on that input, but the activation at the specified site is replaced; downstream computation is then recomputed. In either experiment, only the chosen site is swapped. Other activations follow from the recipient input and the patched computation.

The paper How to use and interpret activation patching, by Stefan Heimersheim and Neel Nanda, develops these two directions and shows why they can disagree.

How to use and interpret activation patching

Read the paper’s account of noising and denoising to connect each direction of patching to the question it tests.

In Section 2.3, “Noising and Denoising,” read the two directions, beginning with the distinction between source and destination runs. Then read Section 2.4, “Example: AND gate vs OR gate,” from the gate example. Focus on what remains clean or corrupted outside the patched component. In the OR discussion, judge the stated result from the gate logic: noising an individual input leaves the other clean input available.



Conditional necessity and conditional sufficiency

Noising tests conditional necessity for maintaining the clean behavior. If the clean run exhibits the behavior, but replacing the site with its corrupted activation makes that behavior fail, the clean value at that site was needed given this clean recipient, this replacement, and the other activations left available. It does not follow that the component is indispensable on every input or under every possible replacement.

Denoising tests conditional sufficiency for restoring the behavior. If the corrupted run lacks the behavior, but inserting the clean activation makes it appear, that clean activation was sufficient given this corrupted recipient and its available downstream computation. The activation did not produce an output by itself. It supplied something the remainder of the patched run could use.

These are claims about values under interventions, not automatic semantic labels for components. Replacing an entire residual-stream vector, for instance, may transfer multiple signals. A successful patch does not by itself identify which signal mattered.

A compact decision rule is useful. Assume the unpatched clean run meets the behavior criterion and the unpatched corrupted run does not:

Observed result at the specified site Supported interpretation
Noising breaks clean behavior; denoising restores it Both conditional necessity and conditional sufficiency
Noising breaks clean behavior; denoising does not restore it Conditional necessity only
Noising does not break clean behavior; denoising restores it Conditional sufficiency only
Neither patch changes the behavior as specified Neither is supported by these tests

The last row is not proof that the site has no role. The behavior might depend on it only in other contexts, a backup might compensate, or the chosen donor activation might fail to test the distinction of interest. Likewise, a score can move without crossing your declared behavioral threshold; report that movement rather than silently changing what “restored” or “broken” means.



Why the two directions disagree

The AND/OR analogy from the reading makes the conditioning explicit. Suppose the clean state has and , the corrupted state has and , and the behavior is that the output equals . We patch only . Denoising evaluates the output with ; noising evaluates it with .

Output rule Denoising : clean value amid corrupted context Noising : corrupted value amid clean context Result for
AND Necessity only
OR Sufficiency only
Both
Neither

In the AND circuit, restoring cannot rescue the corrupted run while remains off. But turning off in the clean run breaks the output. In the OR circuit, restoring rescues the corrupted run, yet turning it off in the clean run leaves to sustain the output. The two tests are not mirror images because their recipient contexts differ.

This also clarifies the previous lesson’s toy result. Its reference score was , its counterfactual score was , and patching the cue position in either direction moved the score to the donor run’s value. If the specified behavior is a positive fixed logit difference favoring answer A, clean-to-counterfactual patching restored it, while counterfactual-to-clean patching broke it. That supports both labels for that position, paired inputs, replacement, and behavior. It does not establish that the cue activation is universally necessary or sufficient in a learned model.

The paper extends this logic from gates to a stylized multi-component computation. Its example is useful because an intervention late in a computation can restore the result without separately restoring the earlier components that produced the donor activation.

How to use and interpret activation patching

Return to the same paper for a worked interpretation and the limits of the resulting claim.

In Section 3.1, “Walkthrough a stylized example,” read the walkthrough. Track why restoring a later neuron can work when restoring an earlier component alone does not. Then, in Section 3.2, “Concepts & gotchas,” read the qualifications, especially “Sensitivity & prompt choice,” “Scope of activation patching,” and “Backup behaviour & OR-gates.” Relate each qualification to the recipient context in the two patch directions.



State the claim at the resolution of the experiment

A strong interpretation preserves the details that made the result possible. For example:

On these matched prompts, replacing the clean activation at the selected position with its corrupted counterpart made the fixed answer-preference criterion fail. This supports conditional necessity of the clean activation for maintaining that preference under this replacement.

If the reverse patch also restores the criterion, add the corresponding conditional-sufficiency claim. If it only improves the score partway, say that it partially restores the margin instead. This avoids treating a visible numerical shift as proof of a stronger binary claim.

There are three particularly important boundaries around either label.

First, the claim is local to the input contrast. When a pair changes which answer is correct, a fixed answer-A-versus-answer-B score tests how information about that change travels through the model. It does not automatically test reliance on an irrelevant shortcut cue. For a shortcut claim, the paired inputs must preserve task-relevant evidence while changing the suspected nuisance cue, and the behavior metric must still refer to the same task goal.

Second, the replacement is part of the claim. A corrupted activation that breaks a clean run is evidence about that particular substitution. It may carry an alternative answer, disrupt several signals at once, or combine unusually with the recipient’s remaining activations. Conversely, a failed noising test can conceal an active contribution if another pathway compensates. Neither outcome alone licenses “this component is the concept” or “this component does nothing.”

Third, a patched run is not the unmodified run. Denoising demonstrates that downstream computation can use the supplied activation in that recipient context. Noising demonstrates what happens when one value is exchanged while the rest of a clean computation remains available. To claim a fuller mechanism, researchers must test whether the proposed information and interactions explain behavior beyond one successful substitution.

For reporting, keep the unpatched clean and corrupted scores beside both patched scores. Name the behavioral threshold, the site and token position, the donor and recipient for each patch, and the input distribution tested. Those details let a reader tell whether the evidence supports necessity, sufficiency, both, or neither without relying on the visual impact of a patching plot.



Key takeaways

Noising asks whether replacing a clean activation breaks a behavior in a clean recipient: a positive result supports conditional necessity. Denoising asks whether inserting a clean activation restores a behavior in a corrupted recipient: a positive result supports conditional sufficiency. The conditions—behavior criterion, matched inputs, site, replacement, and recipient context—are part of each conclusion. AND-like dependencies and OR-like backups explain why the directions often disagree.

The next lesson moves from swapping naturally occurring activations to intervening along a chosen activation direction, where sham and magnitude-matched controls will help distinguish a targeted effect from a generic perturbation.